We prove the convergence of the extremal processes for variable speed branching Brownian motions where the "speed functions", that describe the time-inhomogeneous variance, lie strictly below their concave hull and satisfy a certain weak regularity condition. These limiting objects are universal in the sense that they only depend on the slope of the speed function at $0$ and the final time $t$. The proof is based on previous results for two-speed BBM obtained in a recent paper of ours and uses Gaussian comparison arguments to extend these to the general case.